Non-uniform correction illumination pattern

ABSTRACT

An example system in accordance with aspects of the present disclosure includes a controller and a camera communicatively coupled to the controller to capture an image of a work surface. A projector is also provided and is coupled to the controller to project an illumination pattern onto the work surface during image capture by the camera.

BACKGROUND

When using a camera to take a photograph, there must be sufficientlighting. Some cameras have built-in flashes while other cameras useexternal lighting sources. Insufficient lighting can render theresulting photograph less than desirable.

BRIEF DESCRIPTION OF THE DRAWINGS

For a detailed description of various examples, reference will now bemade to the accompanying drawings in which:

FIGS. 1A and 1B are perspective, exterior views illustrating one exampleof a projection capture system;

FIG. 2 is a perspective, interior view illustrating one example of aprojection capture system;

FIG. 3 illustrates one example of the camera in the projection capturesystem;

FIG. 4 illustrates one example of the projector in the projectioncapture system;

FIG. 5 illustrates an example of a profile an light illumination patternas received by the camera;

FIG. 6 illustrates an example of non-uniform correction illuminationpattern as projected by the projector;

FIG. 7 shows an example of a system diagram of the projection capturesystem;

FIG. 8 shows another example of a system diagram of the projectioncapture system;

FIG. 9 shows a calibration process for obtaining a non-uniformcorrection illumination pattern in accordance with an example; and

FIG. 10 shows a method of capturing an image using a non-uniformcorrection illumination pattern in accordance with an example.

DETAILED DESCRIPTION Overview:

A mixed reality system is one in which a projector is used to project animage on a work surface while one or more cameras may be present tomonitor and detect objects placed on or around the work surface by auser. In one mode of usage, a user can view an image (e.g., a document,a photograph, a video, etc.) projected onto the work surface by theprojector and then desire to take a digital photograph of an objectplaced on the work surface. The object could be, for example, aphotograph or a three dimensional (3D) object. References to taking adigital photograph, capturing an image, and acquiring an image aresynonymous. When using one of the system's built-in cameras, theprojector's mode of operation changes from displaying an image(document, photograph, video, etc.) to illuminating the work surface soas to use the projector as a lighting source for the camera. As soon asthe image has been acquired by the camera, the projector reverts back todisplaying the original image. The projector thus functions as a “flash”for the camera during image capture.

If the illumination source (the projector) is configured to project acompletely uniform illumination pattern, the light pattern received bythe camera after the uniform illumination pattern reflects off of thework surface and object(s) located thereon may be non-uniform due tovarious irregularities such as project lens non-uniformity and geometricnon-uniformities resulting from the angle and distance traveled by thelight rays to reach the work surface. The resulting image of the objectas captured by the camera may look different depending on where on thework surface the user places the object. This problem is addressed asdescribed below by calibrating the system to compute a non-uniformcorrection illumination pattern. In general, a uniform illuminationpattern is solid white. That is, each output illumination pixel of theuniform illumination pattern is the same as all other pixels in theimage. In some implementations, the output illumination pixels are eachset to an individually predetermined output intensity.

During the calibration process, the system described below captures animage of a blank work surface using a uniform illumination pattern. Ablank work surface is a work surface with no objects (e.g., photographs,documents, etc.) placed thereon and no image projected onto it by theprojector. The resulting captured image then may be inverted to producea non-uniform correction illumination pattern. The non-uniformcorrection illumination pattern subsequently may be used as theprojected illumination pattern during image acquisition. The non-uniformcorrection illumination pattern is computed in such a way that theirregularities noted above cause the light reflected off the worksurface into the camera's lens to generally be uniform.

The illustrative system described below includes a projector thatprojects an image onto a surface by way of a reflecting mirror. Othersuitable implementations include a direct projector, that is, aprojector that projects light and an image directly onto the viewingsurface rather than by way of a reflecting mirror. Further still, thecamera described may be a component of a communication device such as asmart phone. Moreover, in some implementations, the system may be partan all-in-one computer that comprises, among other things, a camera, aprojector, and a display. The following description pertains to oneimplementation but other implementations are possible as well.

EXAMPLE EMBODIMENTS

FIGS. 1A and 1B are perspective, exterior views illustrating one exampleof a projection capture system 10 and an interactive workspace 12associated with system 10. FIG. 2 is a perspective view illustrating oneexample of a projection capture system 10 with exterior housing 13removed. Referring to FIGS. 1A, 1B, and 2, projection capture system 10includes a digital camera 14 and a projector 16. Camera 14 is usable,for example to capture an image of an object 20 in workspace 12 andprojector 16 may be used to project an object image 22 into workspace12. Camera 14 may be a color camera. In some examples, camera 14 isusable to capture an image of the projected object image 22. The lowerpart of housing 13 includes a transparent window 21 over projector 16(and infrared camera 30).

In the example shown in FIG. 1A, a two dimensional object 20 (e.g., ahardcopy of a photograph) placed onto a work surface 24 in workspace 12has been photographed by camera 14 (FIG. 2). Object 20 is shown removedto the side of workspace 12, and object image 22 is projected onto thework surface 24. The object image 22 itself can be photographed bycamera 14 (FIG. 2) and/or otherwise manipulated by a user andre-projected into workspace 12. In the example shown in FIG. 1B, a threedimensional object 20 (a cube) placed onto work surface 24 has beenphotographed by camera 14 (FIG. 2) and then removed to the side ofworkspace 12. An object image 22 is projected into workspace 12 wherethe object image can be photographed by camera 12 and/or otherwisemanipulated by a user and re-projected into workspace 12.

In one example implementation for system 10, projector 16 is configuredto project object image 22 into the same position in workspace 24 as theposition of object 20 when its image was captured by camera 14. Thus, aone-to-one scale digital duplicate object image 22 of an object 20 canbe projected over the original allowing a digital duplicate in its placeto be manipulated, moved, and otherwise altered as desired by a localuser or by multiple remote users collaborating in the same projectedworkspace 12. The projected image can also be shifted away from theoriginal, allowing a user to work with the original and the duplicatetogether in the same workspace 12.

In FIG. 1A, work surface 24 is part of a desktop or other underlyingsupport structure 23. In FIG. 1B, work surface 24 is on a portable mat25 that may include touch sensitive areas. In FIG. 1A, for example, auser control panel 27 is projected on to work surface 24 while in FIG.1B control panel 27 may be embedded in a touch sensitive area of mat 25.Similarly, an A4, letter or other standard size document placement area29 may be projected onto work surface 24 in FIG. 1A or printed on a mat25 in FIG. 1B. Other configurations for work surface 24 are possible aswell. For example, it may be desirable in some applications for system10 to use an otherwise blank mat 25 to control the color, texture, orother characteristics of work surface 24, and thus control panel 27 anddocument placement area 29 may be projected onto the blank mat 25 inFIG. 1B just as they are projected on to the desktop 23 in FIG. 1A.

In the examples shown in FIGS. 1-2, system 10 includes an infrareddigital stylus 28 and an infrared camera 30 for detecting stylus 28 inworkspace 12. The stylus 28 may be battery-operated and rest, when notin use in stylus charging dock 54 (FIGS. 2-4). The stylus 28 may includean infrared light, a touch sensitive nib switch to turn on and off theinfrared light automatically based on touch, and a manual on/off switchto manually turn the infrared light on and off. The infrared light maybe positioned, for example, at or near the tip of the stylus. Infraredlight from the stylus reflects off of mirror 38 and is received anddetected by infrared camera 30 (FIGS. 2-4). Infrared camera 30 is usedto track movement of the stylus 28, thereby permitting the user to movethe stylus about on the work surface while system 10 tracks itsmovement. The stylus can thus be used as a pointing device (similar to amouse), a writing instrument, a drawing instrument, etc.

Although any suitable user input device may be used, a digital stylus(stylus 28) has the advantage of allowing input in three dimensions,including along work surface 24 and without a sensing pad or otherspecial surface. Thus, system 10 can be used on a greater variety ofwork surfaces 24. Also, the usually horizontal orientation of worksurface 24 makes it useful for many common tasks. The ability to usetraditional writing instruments on work surface 24 is advantageous oververtical or mobile computing interfaces. Projecting an interactivedisplay on to a working desktop mixes computing tasks with the standardobjects that may exist on a real desktop. Thus physical objects cancoexist with projected objects. As such, the comfort of using realwriting instruments as well as their digital counterparts (like stylus28) is an effective use model. A three-dimensional pad-free digitalstylus enables annotation on top of or next to physical objects withouthaving a sensing pad get in the way of using traditional instruments onwork surface 24.

Referring to FIGS. 1-4, projector 16 is positioned near base 36 outsideprojector display area 34 (FIG. 4) and focused on mirror 38 so thatlight from projector 16 is reflected off mirror 38 and onto workspace12. Projector 16 and mirror 38 define a three dimensional display space53 in workspace 12 within which projector 16 can effectively displayimages. Projector display space 53 overlaps camera capture space 51(FIGS. 3-4) and is bounded in the X and Y dimensions by display area 34on work surface 24.

Projector 16 may include any suitable light projector. In one example,the projector may be a liquid crystal on silicon (LCOS) projector or adigital light processing projector which is advantageously compact andpower efficient. Projector 16 may also employ a shift lens to allow forcomplete optical keystone correction in the projected image. The use ofmirror 38 increases the length of the projector's effective light path,thereby mimicking an overhead placement of projector 16, while stillallowing a commercially reasonable height for an integrated, standalonedevice.

As explained previously, the projector 16 may serve as the light sourcefor camera 14 during image capturing. Camera capture area 32 (FIG. 3)and projector display area 34 (FIG. 4) substantially overlap on worksurface 24. Thus, a substantial operating efficiency can be gained usingprojector 16 both for projecting images and for camera lighting.

Since projector 16 acts as the light source for camera 12 for imagecapture, the projector light should be bright enough to swamp out anyambient light that might cause defects from specular glare. In someexamples, a projector light of 200 lumens or greater may be sufficientlybright to swamp out ambient light for the typical desktop applicationfor system 10. For still image capture and if the projector is based onlight emitting diode (LED) technology, the projector's red, green, andblue LED's can be turned on simultaneously for the camera flash toincrease light brightness in workspace 12, helping swamp out ambientlight and allowing faster shutter speeds and/or smaller apertures toreduce noise in the image.

As explained above, due to various irregularities involved with theprojector 16 relative to the work surface 24, a uniform illuminationpattern projected by the projector will result in a non-uniform lightpattern as received by the camera 14 after being reflected off of worksurface 24 thereby potentially causing undesirable image capture qualityby camera 14.

In some implementations, the projection capture system may be integratedin or attached to an all-in-one computer, a display, or a tablet device.For example, the projection capture system may be positioned atop avertical support post that also supports an all-in-one computer (i.e., adisplay that also houses the computer's system board) or that supports adisplay. In such implementations, the projection capture system projectsdirectly onto a work surface and/or or touchmat rather than reflectingoff of a mirror.

FIG. 5 shows an example of a light pattern 105 as received by camera 14resulting from a uniform illumination pattern projected initially byprojector 16. The light pattern 105 is what is received by the camera14, and is not the illumination pattern projected by the projector 16.The projector 16, in the example of FIG. 5, projected an illuminationpattern that was substantially uniform (i.e., every pixel has asubstantially similar illumination level). The received pattern 105 hasbeen normalized to an illumination level between 0 and 1 as indicated bythe vertical axis. The example light pattern 105 has a maximum value ofapproximately 1 as indicated by reference numeral 110. The maximum valueat point 110 represents the point in the captured image that has thehighest signal reception and may be the closest point to the projector16. Away from point 110, the illumination level decreases in both the Xand Y directions as shown.

Referring to FIG. 6 and in accordance with the disclosed principles, thesystem 10 performs a calibration process by which a non-uniformcorrection illumination pattern 130 is computed. The non-uniformcorrection illumination pattern 130, such as that shown in the exampleof FIG. 6, is the pattern that is projected by projector 16 (not thepattern actually received by camera 14). The correction illuminationpattern is computed in such a way that upon its reflection off of worksurface 24, the light pattern received into the camera 14 issubstantially uniform. Because the camera receives a substantiallyuniform illumination pattern, image capture is improved relative toimage capturing that would have resulted with the light pattern of FIG.5.

The Hardware:

FIG. 7 shows an example in which system 10 includes a controller 18coupled to the camera 14 and projector 16. A storage device 15 is alsoprovided for storing the non-uniform correction illumination pattern 17for subsequent use during image capture by camera 14. During imagecapture, the controller 18 retrieves the non-uniform correctionillumination pattern 17 and causes the projector 16 to project thenon-uniform correction illumination pattern 17 while camera 14 acquiresthe image.

FIG. 8 provides additional detail of another example of the projectioncapture system 10. The projection capture system 10 includes the camera14, projector 16, and controller 18 as noted above. The system 10 alsoincludes the mirror 38 and a user input device 26. The user input device26 may include the digital stylus 28. The controller 18 may include aprocessor 42, memory 44, and an input/output device 46. The memory 44 isany suitable type of non-transitory computer-readable storage devicesuch as volatile storage (e.g., random access memory), non-volatilestorage (e.g., hard disk, optical disc, etc.), or combinations of bothvolatile and non-volatile storage devices. Memory 44 may include codethat is executable by processor 42 to implement some or all of thefunctionality described herein. Thus, any function described herein asattributed to system 10 is implemented or controlled by controller 18and, more specifically in the example of FIG. 8 is implemented byprocessor 42 executing software module(s) stored on memory 44. In otherimplementations of projection capture system 10, the controller 18 withprocessor 42 may be implemented as a state machine in an applicationspecific integrated circuit (ASIC).

The input/output device 46 may receive information from or sendinformation to an external device. Such information, for example, may beinformation to be displayed on work surface 24, or acquired images to betransmitted to an external device.

Projection capture system 10 in FIG. 8 may also include an objectrecognition device (ORD) 104 for distinguishing between real and virtualobjects in the workspace. In the example shown, object recognitiondevice 104 includes an infrared camera 106, an infrared light 108, and adepth sensor 109. In some examples, a combination of infrared camera106, camera 14, and depth sensor 109 may be used to detect the absenceof physical objects on the work surface 24. The depth sensor 109indicates when a 3D object is on the work surface. The infrared cameraindicates when a thin object (e.g., photos, paper, etc.) are on the worksurface. The camera 14 (which may be a color camera) indicates when thework surface contains color content when the projector projects white.The object recognition device 104 may be used to detect the presence orabsent of a real object on work surface 24. This ability is useful inthe calibration process described below as the calibration processshould be performed without any real objects on the work surface.

The infrared light 108 may be used with camera 106 to illuminate theworkspace to improve object recognition. Also, while it may be possibleto use the same infrared camera for both object recognition (camera 106)and for sensing an IR stylus (camera 30 in FIGS. 2-4), it is expectedthat the camera frame rate for object recognition may not need to be ashigh as the frame rate for sensing stylus position but may requirehigher resolution. Consequently, it may be desirable for someimplementations to use separate infrared cameras for object recognitionand stylus sensing.

Calibration Process:

The calibration process for computing the non-uniform correctionillumination pattern will now be described. The calibration process maybe performed whenever no physical objects are placed on the work surface24, and no content is being projected by projector 16 onto the worksurface, that is, the work surface is blank. A user of system 10 maymanually trigger the performance of the calibration process using thedigital stylus 28 by selecting a “calibration start” button displayed onthe work surface or by other suitable means. Alternatively oradditionally, the object recognition device 104 may be used toconstantly or periodically detect when the work surface 24 is blanks andwhen the work surface is determined to be blank, the object recognitiondevice 104 may trigger the controller 18 (e.g., send a signal to thecontroller) to perform a calibration process. The calibration processthus may be performed multiple times to compute the non-uniformcorrection illumination pattern. The non-uniform correction illuminationpattern may change depending on various factors such as ambient lightingconditions.

The illustrative calibration process of FIG. 9 may be performed bycontroller 18. The operations shown may be performed in the order shownor in a different order and two or more of the operations may beperformed concurrently rather than serially. Upon determining worksurface 24 to be blank (210) as described above using the objectrecognition device 104, the calibration process includes controller 18causing the camera 14 to acquire (214) an image of the blank worksurface using a uniform illumination pattern. During the imageacquisition of the blank work surface, whatever image the projector 16was otherwise projecting is temporarily suspended and the projector 16is caused to project the uniform illumination pattern. Due to theirregularities involved in system 10 relative to the work surface 24,the resulting acquired image may have a non-uniform pattern such as thatshown in the example of FIG. 5.

At 216, the acquired image is normalized (e.g., placed on a scale of 0to 1). At 218, the normalized image is inverted to compute thenon-uniform correction illumination pattern. In one example, theinversion of the normalized acquired image is computed by subtractingeach pixel of the normalized acquired image from 1 (assuming thenormalization results in each pixel being in the range of 0 to 1). Oncethe non-uniform correction illumination pattern is computed, the patternis saved to memory 44 for subsequent use in acquiring an image by camera14.

Image Acquisition:

FIG. 10 shows an example of a method for using camera 14 to acquire animage using the non-uniform correction illumination pattern. The methodmay be initiated manually by a user, for example, by a user selecting an“image capture” button displayed on work surface 24 by projector 16.

At 202, the method includes the controller 18 retrieving the non-uniformcorrection illumination pattern from memory 44. At 204, the methodfurther includes projecting the non-uniform correction illuminationpattern. During operation 204, whatever image was already beingprojected by projector 16, that image's projection is temporarily issuspended in favor of the projection of the non-uniform correctionillumination pattern. The image is then captured by camera 14 at 206.Once the image has been captured, the controller may again cause theprojector to revert back to projecting whatever image was beingprojected before image capture occurred.

The above discussion is meant to be illustrative of the principles andvarious aspects of the present disclosure. Numerous variations andmodifications will become apparent to those skilled in the art once theabove disclosure is fully appreciated. It is intended that the followingclaims be interpreted to embrace all such variations and modifications.

What is claimed is:
 1. A projection capture system, comprising: acontroller; a camera communicatively coupled to the controller tocapture an image of an object on a work surface; and a projectorcommunicatively coupled to the controller to project a non-uniformcorrection illumination pattern onto the work surface during imagecapture of an object by the camera.
 2. The projection capture system ofclaim 1, wherein the controller is to cause the camera to capture animage of a blank work surface during a calibration process and todetermine the non-uniform correction illumination pattern based at leastin part on the captured image of the blank work surface.
 3. Theprojection capture system of claim 2, wherein the controller is furtherto determine the non-uniform correction illumination pattern byinverting the captured image of the blank work surface.
 4. Theprojection capture system of claim 2, wherein the controller is furtherto determine the non-uniform correction illumination pattern bynormalizing the captured image of the blank work surface to produce anormalized captured image and subtracting each pixel of the normalizedcaptured image from
 1. 5. The projection capture system of claim 2,wherein the controller is further to store the non-uniform correctionillumination pattern in memory and subsequently to retrieve thenon-uniform correction illumination pattern from memory for projectiononto the work surface during image capture of the object.
 6. Theprojection capture system of claim 2, wherein the controller repeatedlyis to capture the image of the blank work surface and to store a digitalrepresentation of each non-uniform correction illumination pattern inmemory.
 7. The projection capture system of claim 1 wherein theprojection capture system is to attach to at least one of an all-in-onecomputer and a display.
 8. The projection capture system of claim 1wherein the non-uniform correction illumination pattern comprises aplurality of output illumination pixels being set to an individuallypredetermined output intensity.
 9. A projection capture system,comprising: a controller; a projector communicatively coupled to thecontroller; and a camera communicatively coupled to the controller tocapture an image of an object on a work surface; wherein the controlleris to perform a calibration cycle in which the controller is todetermine that no object is present on a work surface, cause the camerato capture an image of the blank work surface using a uniform lightsource generated by the projector, and generate a non-uniform correctionillumination pattern based at least in part on the captured image of theblank work surface.
 10. The projection capture system of claim 9,wherein the controller is further to generate the non-uniform correctionillumination pattern by inverting the captured image of the blank worksurface.
 11. The projection capture system of claim 9, wherein thecontroller is further to generate the non-uniform correctionillumination pattern by normalizing the captured image of the blank worksurface to produce a normalized captured image and inverting thenormalized captured image.
 12. The projection capture system of claim11, wherein the controller is further to invert the normalized capturedimage by subtracting each pixel of the normalized captured image from athreshold value.
 13. The projection capture system of claim 12, whereinthe threshold value is
 1. 14. The projection capture system of claim 9,further comprising an object recognition device to determine whether anobject is present on the work surface.
 15. The projection capture systemof claim 14, wherein the controller is to perform the calibrationprocess upon the object recognition device determining no object ispresent on the work surface.
 16. The projection capture system of claim9, wherein the projection capture system is to attach to at least one ofan all-in-one computer and a display.
 17. A method, comprising:retrieving a non-uniform correction illumination pattern from a storagedevice; projecting said non-uniform correction illumination pattern; andcapturing an image of an object while projecting said non-uniformcorrection illumination pattern.
 18. The method of claim 17, furthercomprising generating the non-uniform correction illumination pattern byacquiring an image of a blank work surface using a uniform illuminationpattern.
 19. The method of claim 18, wherein generating the non-uniformcorrection illumination pattern comprises inverting the image of theblank work surface.
 20. The method of claim 18, further comprisingdetermining, using a camera, whether the work surface is blank andacquiring the image of the blank work surface using the uniformillumination pattern in response to determining that the work surface isblank.